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Increasingly, educators from all disciplines are using blogs, social networking sites, VLEs and wikis to 
encourage academic discourse between students. However, a common problem experienced by educators is 
how these important learning expediences cadi be assessed and because of this difficulty many are not assessed. 
For some time now, I have been using online discussions via the University VLE as a way to encourage 
student debate around key lecture topics (e.g. Taylor, 2002). The key learning outcomes which this assessed 
activity addresses, in addition to learning more about the topic, are to develop skills in reflective practice, 
critical evaluation and. leadership. This article will review the ivays that face-to-face and online academic 
discourse between students have been assessed, highlighting some of the differences to consider when setting 
up online discussion activities, compared to face-to-face discussion. I will then provide a case study of the ioay 
I set up online discussions and the method I currently use to assess contributions. The final part of the paper 
will consider the potential for using quantitative content analysis (QCA) and automated methods to assess 
online participation. 
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1. Introduction 

NLINE GROUP DISCUSSIONS 
develop many important traditional 
and 21st century skills important for 
graduates in this fast changing world 
(McGraw, 2009). However, existing models 
of assessment typically fail to address these 
skills. This Introduction will consider the 
similarities and differences between the 
learning experiences in face-to-face and 
online group discussions and consider how 
the setting-up of online discussions can 
affect the development of skills. 

1.1 Assessing ‘traditional’ and online discourse 
between students 

Academic discourse between students helps 
them to develop an understanding of 
different views on a topic and helps to 
develop public speaking and debating skills; 
however the learning that takes place during 
such interactions is rarely assessed. 


Although, students may be assessed individu¬ 
ally for their presentation skills or for their 
contribution towards a group project, their 
actual discourse is rarely assessed, due 
mainly to the difficulty in assessing commu¬ 
nication in real time face-to-face environ¬ 
ments. Even if such interactions are 
recorded via video or audio, the complexity 
of such interaction makes assessment excep¬ 
tionally time-consuming and difficult. 

With discourse in online contexts, a 
record is made and interactions take place 
serially (rather than multiple parties 
communicating at the same time), and 
therefore it is easier to assess individual 
contributions. However, there is very litde 
guidance for educators for assessing these 
interactions. Vonderwall et al. (2009) high¬ 
light the paucity of research and practical 
advice on how to assess online postings. They 
discuss assessment processes in asynchro¬ 
nous online discussions, but focus on the 
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variety of aspects that ‘could’ be assessed 
such as self-regulation, learner autonomy, 
learning community and student writing 
skills. They conclude that, ‘asynchronous 
online discussions facilitate a multidimen¬ 
sional process of assessment... further 
research is needed to understand what 
assessment strategies or criteria enhance 
assessment and learning’ (p.309). 

In an early paper, Newman et al. (1994) 
compared critical thinking in face-to-face 
and computer-based seminars (where 
students participated in both), and found 
that more new ideas emerged in the face-to- 
face seminars but that more ideas expressed 
in the online seminars were rated as impor¬ 
tant, justified or linked together; indicating a 
difference in quality and quantity. Heckman 
et al. (2002) compared four face-to-face and 
four online group discussions and found 
that the online discussions generated high 
levels of cognitive activity, which were equal 
to or were superior to those identified in the 
face-to-face discussions. However, they 
provide no guidance on how to conduct 
assessment. 

1.2 Setting up online academic discussions 
I have previously identified key factors for 
educators to consider when setting up 
online discussions and which affect partici¬ 
pation and learning (Taylor, 2002). Three 
factors will be considered here which specif¬ 
ically relate to assessment (task type, assess¬ 
ment strategy, and individual differences), 
however other factors (such as group 
composition and instructor involvement) 
should also be considered. 

As with offline learning activities, task 
type can have a significant impact on the 
quality and quantity of student engagement. 
Kanuka et al. (2007) examined the influence 
of different types of communication activity 
on the quality of students’ messages in an 
online discussion and found significant 
differences between the five activities: the 
nominal group technique; debate; invited 
expert; WebQuest and reflective delibera¬ 
tion. They found that the WebQuest and 


debate activities led to the highest quality of 
messages and highlight the similar qualities 
of these two activities; that they were well 
structured, provided clearly defined roles for 
the students and they provoked the students 
to explicitly confront others’ opinions. 
A number of research articles (e.g. Gafni & 
Geri, 2010) have shown that assessing online 
contributions both increases participation as 
well as enhances the quality of academic 
discourse. Swan et al. (2008) found that 
providing students with the assessment 
criteria led to increases in their participation 
and fostered deeper learning. Regarding the 
impact of individual differences, although 
gender and personality have been shown to 
affect preferences for online discussion, they 
did not affect performance. However, an 
individual difference to consider in assess¬ 
ment that has received little investigation in 
online learning is the impact of having 
English as a second language. Similarly, 
students who have communication difficul¬ 
ties such as dyslexia will need to be assessed 
according to the relevant marking guide¬ 
lines. 

2. Case study: My assessment of 
online discussions 

Hazari (2004) identified two ways to assess 
contributions to online discussions: analytic 
marking which involves assigning marks to 
specified criteria, and holistic marking 
where marks are assigned to the whole unit 
of analysis without scoring individual 
criteria. I use the analytical method and the 
unit of analysis that I use for assessment is 
the message; thus each message is evaluated 
using the criteria below. In addition to 
learning more about the topic and encour¬ 
aging extended research, the key learning 
outcomes which the online discussion 
activity addresses are to develop skills in crit¬ 
ical reflection, evaluation and leadership. 
The assignment consists of participation in 
three online discussions and co-ordination 
and leadership of one discussion. Each indi¬ 
vidual message is assessed by hand using 
similar criteria to that used for other forms 
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of academic wridng, specifically whether it is 
analytical and evaluative. In addition, marks 
are awarded for reflection and timeliness of 
research cited. Each message is graded for 
these criteria using a five-point scale, from 
‘basic attempt’ to ‘excellent’. When leading 
an online discussion, students: provide an 
introduction to the key points; respond to 
other group members' questions; motivate 
discussion, and send a conclusion. The 
leader's posts are graded for coordination, 
responding and motivation. Some brief 
discussion of each of these skills highlights 
relevant research and the defining features 
for assessment. 

2.1 Reflection 

Students are increasingly required to reflect 
critically on their learning as part of their 
coursework, however, as Coutinho (2007) 
highlights, teaching and encouraging reflec¬ 
tive practice is problematic in many ways. For 
example, agreement on what constitutes 
reflective practice is vague and the assess¬ 
ment-centred approach to learning in HE 
often focuses reflection on improving the 
reflective writing style rather than on 
reflecting upon learning (metacognition). 
In addition, when reflections are assessed, 
the incentive is to demonstrate knowledge 
and hide ignorance or doubt which is 
counter to Dewey’s (1939) original purpose 
of reflection in which learning is derived 
from analysing mistakes and solving prob¬ 
lems. Seale and Cann (2000) explored the 
way learning technologies were used to facil¬ 
itate reflective thinking in students. They 
illustrated how a small group of students 
engaged with the material through online 
discussion; students were able to make links 
with other learning experiences and to see 
things in different ways. For my online 
discussions, students are encouraged to 
research widely using both academic sources 
as well as sources from the media (as long as 
due consideration is given to their credi¬ 
bility) . This allows important concepts from 
published papers to be illustrated using clips 
from YouTube and the BBC, and web links to 


articles in the mass media. When I assess 
reflection, marks are awarded for links made 
to personal experiences and examples from 
this wider context. 

2.2 Critical evaluation and extended research 
Mason (1991) proposed that the measure¬ 
ment of online transcripts should be based 
on the educational value that they exhibited. 
He broke this down into a number of useful 
questions, for example, whether a message 
built on previous messages, whether the 
participant drew on their own experience, 
whether they referred to course material or 
material outside the course and whether 
they initiated new ideas for discussion. When 
I assess evaluation, marks are awarded for 
questioning and building on previous 
messages and research. The extent and time¬ 
liness of research and resources used are 
recognised in marking, for example, articles 
published within the last two years and for 
research not already covered in lectures. 

2.3 Leadership skills 

In their study on the importance of trust in 
virtual teamwork, Jarvenpaa and Leidner 
(1998) found that teams with high trust 
levels were more capable of dealing with 
uncertainty and complexity than those with 
low levels. Prior to the online discussions, 
students are provided with a handout on the 
characteristics, benefits and problems of a 
virtual team, along with some useful tips for 
effectively managing and creating trust 
within their online discussions. Regarding 
the style of the online discussion, students 
are advised of the need to find a balance 
between social and task-based communica¬ 
tion. The problems of coordinating or 
leading online teams can be significant, and 
leaders are encouraged to respond quickly 
and to include informal comments so that 
the style of discussion is not a series of long 
monologues and assessment identifies 
attempts to encourage interactivity, for 
example, including participant’s names and 
responding directly to them. This has been 
shown to be one of the key indicators 
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demonstrating an awareness of social pres¬ 
ence and community. A leader’s role is crit¬ 
ical at the start and end of a virtual 
discussion (when the definition of the topic 
and time plan are identified), therefore, 
I assess the introduction and summary 
messages separately. Extra marks are 
awarded also for motivating comments and 
students are encouraged to weave the find¬ 
ings of empirical research into responses 
and questions to others. 

3. Potential for using Quantitative 
Content Analysis (QCA) and software to 
assist assessment 

Due to a change to assessment strategy, the 
online discussions will form the only assess¬ 
ment for this unit in the future (currently it 
forms 30 per cent, with 70 per cent coming 
from an exam). Therefore, there is a need to 
provide more detailed feedback to students 
regarding the assessment of their contribu¬ 
tions and I am considering the potential for 
using content analysis or automated tech¬ 
niques as an additional method of assess¬ 
ment. A literature review of this area 
highlighted the methodologies and software 
that could be used, but the limited empirical 
research highlights important factors to 
consider if assessment is based on these 
methods. 

3.1 Quantitative content analysis (QCA) 
Newman et al. (1995) developed a content 
analysis method to measure critical thinking 
in online group discussions and provided 
textual indicators to identify critical and 
uncritical thinking using sets of paired indi¬ 
cator, for example: relevant/irrelevant; 
important/trivial; new ideas/repeating what 
has been said; putting down new ideas/ 
welcoming new ideas. The system certainly 
has potential for use in the assessment of 
online discourse as the use of obvious oppo¬ 
sites should be easy for an educator (not 
experienced in qualitative analysis) to iden¬ 
tify messages that illustrate these extremes. 
Other papers have been published since this, 
but they are not aimed at educators. For 


example, drawing on 19 key studies 
published in the preceding decade, Rourke 
et al. (2001) cover the potential uses and the 
methodological challenges of analysing 
online transcripts using QCA. This classic 
paper provides a comprehensive discussion 
of issues relating to criteria, research 
designs, units of analysis and ethical issues, 
however, it is not easy for the educator to use 
and seems primarily aimed at educational 
technologists and researchers. Indeed in a 
later paper, Rourke and Anderson (2004) 
propose that QCA is still not systematic and 
objective enough to describe academic 
discourse and provides procedures for devel¬ 
oping the validity of a QCA coding protocol 
that is theoretically valid and to establish its 
validity empirically. Many of the articles 
employing QCA in online environments are 
theoretically driven by the Community of 
Inquiry (Col) framework. This framework 
was developed by Garrison et al. (2001) and 
consists of three elements: cognitive pres¬ 
ence; social presence; and teaching pres¬ 
ence. The first two elements can be used to 
further understand the potential use of QCA 
in assessing reflection, evaluation and lead¬ 
ership. 

3.1.1 Critical evaluation and extended research 
Garrison et al. (2001) provided a detailed 
overview of ways to evaluate online tran¬ 
scripts for evidence of critical thinking based 
on his five stages: (i) problem identification; 
(ii) problem definition; (iii) problem explo¬ 
ration; (iv) problem evaluation/applicability; 
and (v) problem integration. While 
Garrison’s stages are useful they need some 
simplification if they are to be used to assess 
online discussions. It is clear that the main 
focus of later work by Garrison et al. (2006) is 
in producing a methodology to systematically 
and rigorously measure cognitive presence in 
online communications. Their work is very 
useful in guiding educators in the adoption, 
design and implementation of online envi¬ 
ronments for learning, but is less useful as an 
assessment tool. Kanuka et al. (2007) used 
the construct cognitive presence to investi- 
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gate the role of critical discourse in distance 
education and examined the influence of 
different types of communication activity on 
the quality of students’ messages in an online 
discussion. Using QCA to analyse messages 
from 19 students in an undergraduate 
course, each message was assigned to one of 
the four categories of cognitive presence. 
While the number of contributions cate¬ 
gorised in the highest phases of cognitive 
presence was low (20.21 per cent), interest¬ 
ingly they found that it was highest during 
activities which were well structured, 
provided clearly defined roles for the 
students and that provoked students to 
explicitly confront others’ opinions. 
Extended research is probably the easiest to 
assess using QCA and to some extent can 
even be partially automated. For example, 
dates can be highlighted to allow easy identi¬ 
fication of recent research and using a list of 
references already used in lectures the extent 
of new research cited by students can also be 
easily identified. 

3.1.2 Reflection 

A study by Mail' and Taylor (2011) set out to 
identify whether students were reflecting 
online and if so, how deeply. A content 
analysis was conducted on discussion nan- 
scripts using the four types of reflective writing 
identified by Hatton and Smith (1995). The 
study found that the level of reflections within 
the postings became deeper over time. For 
example, although more reflection was occur¬ 
ring in the early discussions, the majority of 
the postings were classed according to Hatton 
and Smith (1995) as level 1 (merely reports 
events with no attempt to provide reasons) 
and level 2 (provides reasons, often based on 
personal judgement). While later discussions 
contain deeper, dialogic and critical reflec¬ 
tions, that is, more at Hatton and Smith’s level 
3 (discourse with one’s self, mulling over 
reasons, exploring alternatives). Although this 
study highlighted the method as a potential 
way to categorise reflections, it has not been 
used for assessment, due to time constraints. 


3.1.3 Leadership skills 

A key skill of an online leader is to 
encourage an atmosphere of trust and 
collaboration and this has been linked to the 
concept of social presence. Rourke et al. 
(1999), drawing on the community of 
inquiry work above, have produced a 
template to assess social presence through 
content analysis. The usability of the 
template for educators is enhanced through 
the provision of selections of coded tran¬ 
scripts and inter-rater reliability figures illus¬ 
trate the validity of this template. However, 
this article is of most use for conference 
moderators and researchers as the focus is 
on setting-up and encouraging social pres¬ 
ence, rather than the assessment of this 
factor. 

3.2 Software to assist analysis, assessment and 
feedback 

Over the last five years, there has been an 
explosion in the use of computer software to 
analyse text and there are literally hundreds 
of software products available which can 
assist analysis of online text (e.g. kdnuggets). 
Many packages are developed for use in 
specific fields and contain features appro¬ 
priate to the type of discourse being 
analysed, for example: politics (e.g. Hopkins 
& King, 2010 look at political speeches and 
campaigns); health (e.g. Kim, 2009 evalu¬ 
ated cancer blogs), and those commercially 
available for marketing and advertising. 
There are relatively few packages developed 
for pedagogic use or for use in the social 
sciences. A review of papers published over 
the last two years found that the packages 
used most often in the social sciences 
include: Linguistic Inquiry and Word Count 
(LIWC); QSR NVivo; TAMS Analyzer (Text 
Analysis Mark-up System); ATLAS.ti, Text- 
STAT, Ranks NL. These range in features, 
from those which produce word frequency 
lists and concordances to those with 
powerful search possibilities (e.g. to identify 
regular expressions or phrases). 
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Despite the advances of methods for 
automated content analysis in the field of 
media analysis, most methods are only able 
to highlight and count instances of pre-spec- 
ified words or phrases and we are a long way 
from automated assessment of critical 
thinking, for example. One potential 
package that could be used immediately is 
the free, open-source template NodeXL, 
which makes it easy to develop network 
graphs from data entered within a 
Microsoft® Excel® spreadsheet. This 
package has great potential to provide assess¬ 
ment feedback in a visual format. 

4. Conclusion 

It is clear that assessment methods need to 
be modernised, to reflect the changes in 
learning activities taking place as a result of 
using interactive and collaborative technolo¬ 
gies made possible by Web 2.0. Additionally, 
updated methods need to consider the expe¬ 
riences and expectations of the current 
generation of students (Taylor, 2011). 
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